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Voting System Certification 
Evaluation Report 

Dominion Voting Systems 
ASSURE 1.3 


Introduction 

The Dominion Voting Systems Assure 1.3 Voting System was evaluated for 
certification by the State of Texas on August 22, 2012. 

Recommendation 

The Dominion Voting Systems Assure 1.3 Voting System is not recommended for 
certification, for the reasons presented in this report. 

This recommendation is being made with the observation that prior versions of the 
system are being successfully used to run elections, including in Texas. A variety of 
features introduced in this version bring improvements to the version of the system 
currently in use and so would be a step forward for those counties that use the Assure 
system. 

Further, version 1.3 of the system has been deployed and is being used in some states. 
Therefore it appears reasonable to assume that remedies can be found and the system 
has the potential to be certified in the state of Texas once those remedies are brought 
forth. In an annex to this report a presentation of Ohio’s work to remedy one of the 
issues, the deficiency of the system verification tools, is presented as a possible model 
for resolving that issue. 

The areas of non-compliance identified were: 

1. Insufficient evidence was provided to find the accuracy of the system’s ballot 
scanners in compliance with HAVA (Help America Vote Act) or Texas 
requirements. Further, even in the very limited time available in during the 
exam evidence was found that mark recognition accuracy may be an issue for 
this system. 

a. In one of several problematic tests, of the 16 ballots counted as part of 
the exam, some marks made by examiners were inconsistently read 
and read differently on the different scanners. Some marks were read 
sometimes but not others when the same ballot was feed to the same 
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scanner. The accuracy of the system was of particular concern because 
no details were provided as to how testing was performed to verify that 
the system meet the HAVA accuracy requirements. 

b. Test methods and test cases were not made available for this 

examination despite these having been specifically requested well in 
advance of the exam. The test report that was presented is summative 
in nature, leaving many aspects of compliance testing undocumented. 
In the area of system accuracy this is particularly troubling because the 
test traditionally run by SLI International (formerly SysTest 
Laboratories) is know to allow a variety of flaws to pass through 
undetected and seldom fails any system. 

2. There were disability access issues with the system. 

a. When voting with the audio ballot with the screen blank, as a blind 
voter would vote, and different from voting visually, at the end of 
voting the voter cannot review their selections. They are instructed to 
cast their vote and then receive a summary of their votes. 

b. The audio quality on the TX R6 unit was very poor quality with a great 
deal of noise that made the audio ballot hard to hear. 

3. PCS Central Count had a high rate of paper jams and misfeeds during the 
demonstration. From the performance observed in the exam the unit could not 
be in compliance with the paper handling requirements of the VVSG. 

4. The system audit log and supporting error detection processes were found to 
be deficient. 

a. The PCS Central Count scanner did not have a printed real time audit 
log attached, as required by Texas code. 

b. A method was discovered to subvert the printed audit log on the 
GEMS system, initially preventing audit events from being printed and 
then erasing all record of them. 

c. The log files were only obtainable in printed form, on narrow and hard 
to read paper strips, making review of the logs when any significant 
number of scanners and DRE’s are used is a prohibitively labor 
intensive, manual process and therefore impractical to perform 
routinely. The lack of access to the logs in an electronic form that can 
be automatically scanned for irregularities in a timely manner makes 
the logging system functionally unusable. The VVSG requires that the 
system report system errors so that election officials can know if 
system components failed to operate correctly during an election. This 
system keeps those component failure messages buried in individual 
unit logs and does not bring them forward so that an election official 
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can be aware of any individual unit problems. The only way to know 
if all units operated correctly is to review the individual logs, but those 
are only available in a form that would require an inordinate amount of 
time and effort to perform this necessary review. 


d. The meaning of log file messages was not found in the documentation 
provided. During the exam additional documentation was provided but 
time did not permit it to be reviewed during the exam. However, 
subsequent review failed to find specific documentation on the 
meaning of status and error messages recorded in the audit logs or 
what actions should be taken for any errors found. The failure to 
document the meaning of log messages would leave election officials 
guessing as to the meaning and significance of the messages. 


5. The version of the system currently deployed in Texas has security certificates 
that expire in June 2013 and January 2014, making those systems non¬ 
functional after those dates. The remedy proposed is to ignore dates on 
certificates, which violates good security practice. Security certificates should 
expire, but as a date of the State’s choosing. Updating security certificates 
should not require purchase of new software from the company. 


A related issue is that the security authority, which is a different issue from 
the security certificates, also has a time limit and will expire on a different 
date but in a similar timeframe to the security certificates. This will also 
render current systems inoperable. The fact of this second date expiration was 
not stated and the date of that the security authority will expire, for the 
systems being used in Texas, was given. 

6. An issue with the company’s customer support processes is revealed by the 
fact that the State of Texas only became aware of the expiring certificate issue 
during this exam. States as well as local jurisdictions should be routinely 
notified whenever a company becomes aware of an issue with its system. In 
the case of the expiring certificates this would allow both state and local 
officials the maximum planning horizon to develop mitigation plans. 

7. The ability to verify that the system is unmodified and continues to be in its 
certified state was found to be deficient. 

a. The system verification tools were incomplete and not usable without 
considerable additional work from people with advanced technical 
skills. To further support this finding see the presentation from the 
State of Ohio in Appendix A. 

b. The verification of the precinct scanners requires removing chips and 
reading them in a chip reader. This method is well beyond the 
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technical capabilities of most jurisdictions and not practical or 
desirable as part of any routine verification of the system. 


8. It was revealed in the same that some system components had not been 
produced in the last 6 years and some in the last 10 years. The viability of the 
system to be purchased as a new system was questioned. It was also 
questioned how long the company would be able to service and support 
systems already in use. A related question is what the impact of 6-10 years of 
storage of the units might be on its reliability. 

9. Issues with configuration management, particularly as it relates to 
Commercial-off-the-Shelf (COTS) used in the system were identified. 


a. In the EAC certificate of certification for the system examined one 
option for GEMS is listed as running on a Dell PowerEdge 2900 
service using the Windows XP operating system. This combination is 
not supported by Dell and when contacted, Dell customer service 
expressed concern about the stability of the combination. In 
particularly concern was expressed about the interface to the disk in 
the system. 

b. The Dell PowerEdge 2900, and perhaps other models of COTS listed 
with the certified system, is no longer available for purchase. This 
means that new purchases would have to use replacement models 
which have not been tested at this point by either the EAC or the State 
of Texas. 


Remedial Actions 

The following remedial actions are recommended for addressing some of the issues 
cited: 

1. It has been said that the best evidence that a voting system is ready for 
certification is its ability to run a good election. Because this system is in 
use it would be highly informative to obtain the logs from some systems 
being used in other states. These logs would then be used to determine the 
experienced performance of the system in real elections. 

2. Accuracy testing that evaluates the system’s ability to recognize marks of 
differing color, size and position in the target area are needed in order to 
know the true system accuracy. This is because the real mark recognition 
accuracy of the system depends on how sensitive it is to variables of color, 
mark size and location. 

3. The audit trail is obtainable and usable without an inordinate effort or 
highly specialized technical abilities. It should be easy to get the logs and 
clear as to what their messages mean. This can be accomplished but does 
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not currently exist. The logs do exist as electronic files and all that is 
needed is a utility that would decrypt them and put them into a format that 
is readable by spreadsheet or similar commonly used software. 

The system validation tools can be completed so that election officials can 
routinely verify that the system they are using in an election is unmodified 
from its certified condition. 

The security certificate and certification authority should be set to dates 
the states is aware of and finds acceptable, with an update process that is 
also acceptable to the State of Texas. 



Sincerely, 


H. Stephen Berger 
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Candidate System 

This section describes the candidate system, the Dominion Assure 1.3 Voting System. 

System Components 

The system is comprised of the following components, based on companies 
“Application for Texas Certification of Voting System” (Form 100). 


1 

System Compo 

nents 1 



Unit/Application 1 

■Versio 


1 Function 1 

1 

GEMS 

1.21.6 

Election Management System 

2 

AccuVote-OS (Precinct Count) 
with new Memory Card 

1.96.14 

Precinct Count Ballot Scanner 

3 

AccuVote-OS (Central Count) 

2.0.15 

Central Count Ballot Scanner 

4a 

AccuVote-TSX BallotStation 

4.7.10 

Direct Recording Electronic (DRE) 
Voting Device 

4b 

AccuVote-TS BallotStation 

4.7.10 

Direct Recording Electronic (DRE) 
Voting Device 

5 

TSX WinCE 

410.3.10 

Operating System 

6 

TS WinCE 

300.3.5 

Operating System 

7 

TSX/TS Bootloader 

1.3.11 

Bootloader for TSX and TS 

8 

Key Card Tool 

4.7.8 

Security Key Card Tool 

9 

ABasic 

2.2.5 

Report Scripting 

10 

Voter Card Encoder 

1.3.3 

Voter Access Card Encoder 

11 

VC Programmer 

4.7.8 

Voter Access Card Programmer 

12 

Cardwriter 

1.1.6 

Voter Access Card Encoder 

13 

PCS Central Count 

2.2.5 

Central Count Ballot Scanner 

14 

Assure Security Manager 

1.2.5 

System Security Management 
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System Configuration 


Figure 1 and Figure 2 illustrate typical system configurations. 
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Figure 1 - Assure 1.3 Polling Place and Early Voting Configuration 
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Management System with optional Accu Feed 
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Figure 2 - Assure 1.3 Central Count and Absentee/Mail-in Voting Configuration 
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Examination Report 

The examination started with a description of the Assure 1.3, including its 
configuration and the function and role of the various components in the voting 
system. 

Compliance Checklist 

The following checklist includes all Texas voting system requirements. The complete 
checklist is provided as detailed support for the conclusion and recommendation of 
this report. 


Vendor: Dominion Voting Systems Voting System: Assure 1.3 


Pre-Test Requirements 

• Is Form 100 complete and satisfactory? ^ 

If not satisfactory, please list questions to ask vendor. 

fe 

X 

s No 

n 

• Review Form 100 - Schedule A - Have recommendations/issues made from previous exams been 
corrected or addressed? 

fe 

X 

s No 

n 

• Review Form 101 - Are responses satisfactory? ^ 

fe 

X 

s No 

n 

• Review change logs and provide information for testing or questioning vendor ' 

fe 

X 

s No 

n 

• Training manuals appear complete? Yes No 

□ X 

• Training manuals appear to be easy to use? Yes No 

□ X 

• Check with other jurisdictions where system is in use and ask questions regarding system, support Yes No 

and training. O O 

• Did the system receive favorable reviews? Yes No 

If not, please explain. Q Q 

• Do all configurations listed in application seem feasible? Keep this in mind during the 

examination to make sure components necessary to ensure the security are included in all 1 

configurations and that the configurations will meet the counties needs (scanner used as central 
and/or precinct, etc..) 

fe 

X 

s No 

□ 

• Vendors' proposals shall state a clear, unequivocal commitment that the election management and ^ 

voter tabulation software user's application password is separate from and in addition to any other 
operating system password. 

fe 

X 

s No 

n 

• Vendor's system shall support automated application password expiration at intervals specified by s 

a central system administrator. 

fe 

X 

s No 

n 

• Vendor shall discuss the steps required by the system administrator to implement and maintain 

automated password expiration. This discussion will include narrative concerning the degree to ? 

which the application password expiration capabilities are based on (a) the server or client's 
operating system, (b) the software application, or (c) both 

fe 

X 

s No 

□ 

• The vendor’s proposal shall state the name of any automated incident, issue, or problem tracking " 

system used by the firm in providing support to its election system clients. 

fe 

X 

s No 

n 
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(Note: Technical Bulletins for the previous year were provided and approved.) 


Verify Installation 

• Verify/List all hardware ^ 

fe 

X 

s No 

n 

• Verify/List all COTS hardware/software versions A 

fe 

X 

s No 

n 

• Is the COTS hardware being demonstrated the same version as what was tested at the VSTL? ^ 

fe 

X 

s No 

n 

• Is the COTS software being demonstrated the same version as what was tested at the VSTL? Yes 

□ 

Me 

X 

> 

• Witness or actual install the software and firmware with the SOS CDs received from VSTL. Yes 

□ 

Me 

X 

> 
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Vendor: Dominion Voting Systems Voting System: Assure 1.3 


Texas Federal 

Law Law 


System Review 

TEC • Preserves the secrecy of the ballot ^ 

122.001 

fe 

x 

s No 

□ 

TEC • Is suitable for the purpose for which it is intended Yes ] 

122.001 |—] 

STc 

X 


TEC • Operates safely, efficiently, and accurately and complies with the error Yes No 

122 001 rate standards of the voting system standards adopted by the FEC (EAC) EH EH 

TEC • Is safe from fraudulent or unauthorized manipulation (physical exam and Yes ] 

122.001 n ! , 1—1 

review or manuals) LJ 

STc 

X 


TEC • Permits voting on all offices and measures to be voted on at the election A 

122.001 

fe 

x 

s No 

□ 

TEC HAVA • Warns of Overvote - Prevents counting votes on offices and measures on 1 

122 001 

which the voter is not entitled to vote 

fe 

x 

s No 

□ 

HAVA • Warns of Undervote ' 

fe 

X 

s No 

□ 

TEC • Prevents counting votes by the same voter for more than one candidate 

122 001 

for the same office or, in elections in which a voter is entitled to vote for 1 

more than one candidate for the same office, prevents counting votes for 
more than the number of candidates for which the voter is entitled to vote 

fe 

X 

s No 

□ 

TEC • Prevents counting a vote on the same office or measure more than once ^ 

122.001 

fe 

X 

s No 

□ 

TEC • Permits write-in voting ^ 

122.001 

fe 

X 

s No 

□ 

122001 9 ca P a ^ e P erm ihi n g straight-party voting 1 

fe 

X 

s No 

□ 

TEC • Is capable of cross-over votes ^ 

65.007 

fe 

X 

s No 

□ 

TEC HAVA • j s capable of providing records from which the operation of the voting Yes ] 

1 ““' 001 system may be audited EH 

STc 

X 


• Is it easy to choose the appropriate ballot style? ^ 

fe 

X 

s No 

□ 

• Is the number of ballot styles available on a unit limited? ^ 

fe 

X 

s No 

□ 

• Can you cancel the marking of a ballot after starting? ^ 

Explain how. 

fe 

X 

s No 

□ 

• Is there a way to properly secure all ports on the system? 1 

fe 

X 

s No 

□ 

• Are instructions provided in the documentation for securing the system? 1 

fe 

X 

s No 

□ 

• Usable for curbside voting? 1 

fe 

X 

s No 

□ 

• How to setup or modify audio files Yes No 
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X 

□ 



• 

How to adjust volume 

1 

fe 

X 

s No 

□ 



• 

Does the system have any RF (Radio Frequency) communications? 

Yes 

□ 

] 

NTc 

X 




• 

Have representatives of the visually impaired community evaluated the 
accessibility of the system? 

1 

fe 

X 

s No 

□ 



• 

Test both early voting and election day - all functions opening/closing 


fe 

X 

s No 

□ 



• 

Does system include sip 'n puff for accessibility 


fe 

X 

s No 

□ 



• 

Does system include paddles for accessibility 

1 

fe 

X 

s No 

□ 


DRE Review 

TEC 

122.001 


• 

Preserves the secrecy of the ballot 

1 

fe 

X 

s No 

□ 

TEC 

122.001 


• 

Is suitable for the purpose for which it is intended 


fe 

X 

s No 

□ 

TEC 

122.001 


• 

Operates safely, efficiently, and accurately and complies with the error 
rate standards of the voting system standards adopted by the FEC (EAC) 


fe 

X 

s No 

□ 

TEC 

122.001 


• 

Is safe from fraudulent or unauthorized manipulation (physical exam and 
review of manuals) 


fe 

X 

s No 

□ 

TEC 

122.001 


• 

Permits voting on all offices and measures to be voted on at the election 


fe 

X 

s No 

□ 

TEC 

122.001 

HAVA 

• 

Warns of Overvote - Prevents counting votes on offices and measures on 
which the voter is not entitled to vote 


fe 

X 

s No 

□ 


HAVA 

• 

Warns of Undervote 


fe 

X 

s No 

□ 

TEC 

122.001 


• 

Prevents counting votes by the same voter for more than one candidate 
for the same office or, in elections in which a voter is entitled to vote for 
more than one candidate for the same office, prevents counting votes for 
more than the number of candidates for which the voter is entitled to vote 


fe 

X 

s No 

□ 

TEC 

122.001 


• 

Prevents counting a vote on the same office or measure more than once 


fe 

X 

s No 

□ 

TEC 

122.001 


• 

Permits write-in voting 


fe 

X 

s No 

□ 

TEC 

122.001 


• 

Is capable of permitting straight-party voting 


fe 

X 

s No 

□ 

TEC 

65.007 


• 

Is capable of cross-over votes 

1 

fe 

X 

s No 

□ 

TEC 

122.001 

HAVA 

• 

Is capable of providing records from which the operation of the voting 
system may be audited 

Yes 

□ 

] 

STc 

X 


• Reports available by precinct? ^ 

fe 

X 

s No 

□ 

• In order to perform a manual recount, can you print cast vote records for a ^ 

precinct (including early voting, ED and absentee?) from an individual 

DRE? 

fe 

X 

s No 

□ 

TAC 


• 

A DRE must have the capability to segregate provisional votes from 

Yes 

No 
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81 176 regularly-cast votes on the precinct returns 

X 

□ 

TAC • The precinct returns must indicate the number of provisional ballots cast ^ 

81 176 x A 

but not include actual provisional votes in the unofficial totals from the 

precinct 

fe 

X 

s No 

□ 

TAC • Must provide a method for the cast provisional ballots to be accepted & ^ 

81 176 x x x 

added to the election results 

fe 

X 

s No 

□ 

• Must be designed to not accept provisional write-in votes until the 1 

provisional vote has been accepted/approved. 

fe 

X 

s No 

□ 

TEC • Equipped with a security system capable of preventing operation of the 1 

122.033 t • 

machine 

fe 

X 

s No 

□ 

TEC • Equipped with registering counters that can be secured against access A 

122.033 

fe 

X 

s No 

□ 

TEC • Equipped with a public counter A 

122.033 

fe 

X 

s No 

□ 

TEC • Equipped with a private counter A 

122.033 

fe 

X 

s No 

□ 

TEC • Does each unit have a permanent identification number? A 

127.154 

fe 

X 

s No 

□ 

• Capability to have more than one ballot style available on a machine 1 

(used for consolidated precincts and early voting) 

fe 

X 

s No 

□ 

• Can you easily choose the ballot style used on a DRE? ^ 

fe 

X 

s No 

□ 

HAVA • Provide voters with disabilities the same opportunity for access & Yes ] 

participation (including privacy & independence) PI 

NTc 

X 


• Usability of taking system to curbside voter 1 

fe 

X 

s No 

□ 

HAVA • Allow voter to review selections before casting ballot A 

fe 

X 

s No 

□ 

HAVA • Allow voter to change selections before casting a final vote Yes ] 

□ 

STc 

X 


• Do multiple choice selections appear on summary screen? EX: vote for 2 ^ 

or more 

fe 

X 

s No 

□ 

• Does the system have any RF (Radio Frequency) communications? Yes ] 

□ 

NTc 

X 


• Is there a way to properly secure all ports on the system? ^ 

fe 

X 

s No 

□ 

• Are instructions provided in the documentation for securing the system? 1 

fe 

X 

s No 

□ 

• Have representatives of the visually impaired community evaluated the 1 

accessibility of the system? 

fe 

X 

s No 

□ 

• Test both early voting and election day - all functions opening/closing A 

fe 

X 

s No 

□ 

• Does system include sip 'n puff for low mobility 1 

fe 

X 

s No 

□ 


Texas Real-time Audit Log Review 

TEC • A central tabulating device must include a continuous feed printer y es ^ 

81,62 dedicated to a real-time audit log, which prints out all significant election i—i 

events and their date and time stamps. 

NTc 

X 
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TEC 

81.62 


TEC 

81.62 


TEC 

81.62 


TEC 

81.62 


TEC 

81.62 


TEC 

81.62 


TEC 

81.62 


TEC 

81.62 


See VYSG 2005: 


2.2.5.2.1.d: "The audit record shall be active whenever the system is in an 
operating mode. This record shall be available at all times, though it need 
not be continually visible." 


2.2.5.2.1.g: "The system shall be capable of printing a copy of the audit 
record." _ 

Log error messages and operator response to those messages 

See VVSG 2005 Section 2.2.5.2.2.a & 4.4.3.d _ 

Log the number of ballots read for a given precinct 

See VVSG 2005 Section 4.4.4.a & c & e _ 

Log completion of reading ballots for a given precinct 

See VVSG 2005 Section 4.4.3.b.3 _ 

Log the identity of the input ports used for modem transfers from 
precincts 

See VVSG 2005 Section 4.4.2.g.l-4 _ 

Log users logging in and out from election system 

See VVSG 2005 4.4.3.a.4, 4.4.3.d, 6.5.5.a & c _ 

Log precincts being zeroed 

See VVSG 2005 4.4.3.b.2 _ 

Log reports being generated 

See VVSG 2005 4.4.3.d _ 

Log diagnostics of any type being run 

See VVSG 2005 4.4.2.a & d _ 

Print any attempt to tally or load votes that have already been tallied or 
counted, identifying the precinct or source of the votes and flagging it as a 
duplicate _ 

Print starting the tally software (e.g. from the operating system) or exiting 
the tally software, or any access to the operating system. _ 

Record if a printer is paused, turned off, turned on, disconnected, and 
when reconnected. 


Yes 

□ 

Yes 

□ 

Yes 

□ 


Yes 

□ 


Yes 

□ 

Yes 

□ 

Yes 

□ 

Yes 

□ 

Yes 

□ 

Yes 

n 

Yes 

□ 


No 


No 

□ 


No 

□ 


No 

□ 


No 

□ 


No 

□ 


No 

□ 


No 

□ 


No 

□ 


No 

□ 


No 


TEC 

122.001 


TEC 

122.001 


TEC 

122.001 


Optical Scan System Review 

Preserves the secrecy of the ballot 


Is suitable for the purpose for which it is intended 


Operates safely, efficiently, and accurately and complies with the error 
rate standards of the voting system standards adopted by the EAC 


Yes 


Yes 

□ 


Yes 

□ 


No 

□ 


No 


No 
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TEC • Is safe from fraudulent or unauthorized manipulation (physical exam and Yes No 

122 001 review of manuals) 1 1 X 

TEC • Permits voting on all offices and measures to be voted on at the election ^ 

122.001 

fe 

x 

s No 

□ 

TEC HAVA • Warns of Overvote - Prevents counting votes on offices and measures on 1 

122 001 

which the voter is not entitled to vote 

fe 

X 

s No 

□ 

HAVA • Warns of Undervote ' 

fe 

X 

s No 

□ 

TEC • Prevents counting votes by the same voter for more than one candidate 

122 001 

for the same office or, in elections in which a voter is entitled to vote for 1 

more than one candidate for the same office, prevents counting votes for 
more than the number of candidates for which the voter is entitled to vote 

fe 

X 

s No 

□ 

TEC • Prevents counting a vote on the same office or measure more than once ^ 

122.001 

fe 

X 

s No 

□ 

TEC • Permits write-in voting ^ 

122.001 

fe 

X 

s No 

□ 

122001 # ca P a ^ e P erm ihi n g straight-party voting A 

fe 

X 

s No 

□ 

TEC • Is capable of cross-over votes A 

65.007 

fe 

X 

s No 

□ 

TEC HAVA • j s capable of providing records from which the operation of the voting 1 

122 001 system may be audited 

fe 

X 

s No 

□ 

• Reports available by precinct? ^ 

fe 

X 

s No 

□ 

• In order to perform a manual recount, can you print cast vote records for a ^ 

precinct (including early voting, ED and absentee?) from an individual 

DRE? 

fe 

X 

s No 

□ 

TEC • Does each unit have a permanent identification number? 1 

127.154 

fe 

X 

s No 

□ 

• Is there a way to properly secure all ports on the system? 1 

fe 

X 

s No 

□ 

• Are instructions provided in the documentation for securing the system? ^ 

fe 

X 

s No 

□ 
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Supplemental Discussion 



Complex History 

The testing of this system suffered an unusually complex and difficult history. The 
system started its certification testing as the Assure version 1.2 voting system at 
SysTest Labs (now operating as SLI International) in mid-2007. It progressed 
through the certification testing process until the summer of 2008. On October 29, 
2009 Premier Voting Systems, the company that originally developed the Assure 
system, requested EAC permission to move the project to iBeta Laboratory. They 
cited as their reason the imminent deaccreditation of SysTest Laboratories (now 
operating as SLI International). The testing was completed by iBeta Laboratories and 
the Assure 1.2 system was granted certification on August 6, 2009 by the EAC. 
However, shortly thereafter some problems were discovered with the system, which 
the company then addressed. They returned the modified version of the system, now 
the Assure 1.3 system to SLI Laboratories (Formerly SysTest labs, but operating 
under new management and with extensive changes in technical personnel.) The 
Assure 1.3 system was certified by the EAC on June 29, 2012. Essentially the 
certification testing was divided between 3 laboratories and performed over a 5 year 
period. 

Many of the early tests were not required to be rerun, but were accepted for reuse in 
the certification process. However, some of these tests were performed at SysTest 
shortly before its accreditation was revoked due to significant quality problems. The 
newest testing on the Assure 1.3 system focused on the modifications and much of 
the testing on the Assure 1.2 system was accepted for reuse. However, during this 
time deficiencies were identified in some test methods, notably the accuracy testing 
and the EAC worked with the labs to improve their test methods in various areas. 
However, SLI International was disengaged from the process during much of this 
time. 

Further complicating the process Premier Voting Systems was acquired by ES&S. 
ES&S was then required by the US Department of Justice to divest itself of certain 
parts of the company. Portions of Premier were then sold by ES&S to Dominion 
Voting Systems. It is Dominion Voting Systems which is applying for Texas 
certification of the system. 


Insufficient Description of Testing 

This unusually complex history makes review of the testing even more critical than it 
normally would be. Testing of any voting system is complex and a state review of 
testing is always advisable even though the EAC is both careful in its work and 
technologically skilled. With a history as complex as that of the Assure 1.3 system, 
careful review is particularly in order. However, SLI International has taken the 
position that the specific test cases are proprietary and will not be disclosed to the 
State of Texas. Without that level of detail it is not possible to perform an 
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independent review of the testing performed. All that can be said is that SLI asserts 
the system meets all requirements of the VVSG. However, it is relying largely on 
testing by SysTest, which was deaccredited, and iBeta, which has now withdrawn 
from the business. This position that test methods are proprietary is highly unusual in 
testing of this type. Normally the methods and specific detail of testing performed for 
certification or regulatory purposes are fully disclosed, allowing independent review 
of them. Laboratory tests are expected to be documented with sufficient detail so that 
either another laboratory can repeat a test to verify the reported result or that other 
engineers can form an independent assessment of the testing performed. With the 
high level test report provided by SLI Laboratories no independent assessment can be 
made of the quality or appropriateness of the testing. As a result it must be concluded 
that there is not sufficient evidence provided in this exam to support a finding of 
compliance with many requirements, most specifically with the system accuracy 
requirements. 


System Accuracy 

System accuracy is a central requirement for any voting system. HAVA includes the 
following requirements regarding system accuracy: 

(5) Error rates.--The error rate of the voting system in 
counting ballots (determined by taking into account only 
those errors which are attributable to the voting system 
and not attributable to an act of the voter) shall comply 
with the error rate standards established under section 
3.2.1 of the voting systems standards issued by the 
Federal Election Commission which are in effect on the 
date of the enactment of this Act. 

(6) Uniform definition of what constitutes a vote.-Each State 
shall adopt uniform and nondiscriminatory standards that 
define what constitutes a vote and what will be counted 
as a vote for each category of voting system used in the 
State. 

The accuracy required in the VVSG is: 

A target error rate of no more than one in 10,000,000 ballot 
positions, with a maximum acceptable error rate in the test 
process of one in 500,000 ballot positions. 

All paper ballot scanners will have problems with some marks. Color, size and 
position of marks are important variables when evaluating mark recognition accuracy. 
It is important for states to know the real accuracy of voting systems, evaluated 
against the range of marks made by voters, in order to craft election procedures that 
match the capabilities of the specific system being used. It is extremely important 
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that election officials know the kinds of marks that a system will find problematic so 
that they can effectively deal with close elections. 

The accuracy tests used historically have been shown to be flawed, allowing deficient 
systems to pass the test. The EAC has worked with the labs to improve testing in this 
area and work in this area is ongoing. Given the long history of this particular system 
and the fact that test methods have improved in this area, it is very important that the 
test methods used to evaluate the accuracy of the system be known and in all 
likelihood new accuracy testing will be required if the State of Texas is to know the 
true accuracy of the system, measured against the range of marks typically made by 
voters, particularly on absentee, mail-in ballots. 

Configuration Management 

In the EAC certificate of certification for the system examined one option for GEMS 
is listed as running on a Dell PowerEdge 2900 service using the Windows XP 
operating system, Figure 3. However, the PowerEdge 2900, being a serve was not 
designed for or intended to be used with consumer type operating systems such as 
Windows XP. 


System Component 

Software or 

Firmwa re Version 

Hardwa re 

Version 

Operating System or 

COTS 

Comments 

Ballot Preparation and Central Count 

GEMS 

1.21.6 


Dimension 3100 

Windows XPSP3 

PowerEdge 2900 
Windows XP SP3 

PowerEdge 1B00 
Windows Server 2003 

SP2 



Figure 3 - On the EAC Certificate of Conformance a non-supported and deprecated 

configuration of the Dell PowerEdge 2900 is listed 1 


Dell only lists the following operating systems for use with the PowerEdge 2900: 

• Microsoft® Windows® Server 2003 R2, Standard, Enterprise Edition, x64, 
Standard and 

• Enterprise Edition; Microsoft® Windows® Server 2003 Small Business 
Standard, Premium 

• Edition; Microsoft® Windows® Storage Server 2003 R2, Standard, 
Enterprise Edition; 

• Red Hat® Linux® Enterprise v4, ES EM64T; 




EAC Certificate of Conformance for the DVS Assure 1.3 Voting System, dated June 29, 2012. 
http://www.dell.com/downloads/emea/products/pedge/en/PE2900_Spec_Sheet_Quad.pdf 
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• SUSE Linux Enterprise Server 9 EM64T 



When contacted, Dell customer support stated that they would have concern about the 
stability of Windows XP operating on a PowerEdge 2900 because neither they nor 
Microsoft make a driver for the disk interface. The use of an unknown driver raises 
questions about the stability and integrity of the data being written to the disk. 


The presence of this example of COTS being modified, particularly in such a critical 
area, but still being reported as unmodified and commercially available raises 
questions about the company’s quality and configuration management systems. 


The problem is further complicated by the fact that the Dell PowerEdge 2900 is no 
longer available, Figure 4. When the issue of model obsolescence was discussed with 
Dell product management they expressed interest in working with voting system 
manufacturers to identify models which would have significant market life, help them 
identify appropriate replacement models and even work with them to perform some 
testing. However, it appears that Dominion is not working with its COTS providers 
at this level. This brings into question their supply chain management and increases 
the likelihood of election officials suffering the consequences as products are 
withdrawn, if replacement products do not function in the same way. The concerns 
created by the mismatched operating system and computer become even greater since 
the computer that will be used with the system is unknown. 



Outlet Products 


Laptops v 


Desktops v 


Workstations v 


Servers 
& Storage 


Dell Outlet Business & Education > Outlet Servers & Storage > Servers > Tower Server 


Dell PowerEdge 2900 Server Details 

★★★★★ (63 Ratings) write a review 



This product is unavailable. 
Below we have suggested a 
comparable system. 


Figure 4 - Dell PowerEdge 2900 is no longer available less than 6 weeks after the 

system’s EAC certification 3 


O 

http://www.dell.com/us/dfb/p/poweredge-2900/pd#TechSpec 
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Annex A - OHIO Verification Experience 

In June of 2012 the following report was presented of Ohio’s experience with verifying the Assure 1.2 system. It is included here both 
to support the findings of this exam but also because it points toward solutions that Ohio developed with the system. 
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Checking the List Twice 


State Certification Testing of Voting Systems 

National Conference 

Indianapolis, Indiana 
June 14-15, 2012 

Joshua Franklin 
Matthew Masterson 



Danielle Sellars 
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What. When, and How to Verify? 



Relevant Fads 
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Purpose 


* To explain our experiences in verifying the 
physical, software, and set up configuration 
for the voting systems in Ohio's 88 
counties. 
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Team Effort 


• Accomplishing this is a bumpy road 

• Required federal, state, and local efforts 

• Danielle Sellars provided the footwork and 
onsite technical know-how 
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Why Verjfy.7_ 

• Keep the system safe, secure, and 
certified. 

• Software is the same during distribution, 
installation, setup. [1] 

• Supports a chain of custody 

• “Software integrity: ensuring that the 
software programs have not been altered, 
whether by an error, a malicious user, or a 
virus.” - Bruce Schneier 
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When to Verify? 


There is no single answer: 
At time of installation? 

L&A 

Before the election? 


After an election? 

After canvass? 

Part of post-election audit? 
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X 




What and How to Verify? 

Check the: 

- Installation media 

- Software already on the machine 

- Documentation 

System Identification Tools from 
manufacturer 

- Validate the hashes of the static software files 

- Provides high level of assurance that the 
software is unchanged 
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1 




Relevant Facts 


* Purchased in 2002 

- Systems have never validated 

- Numerous upgrades to fielded system since 
then 

* OH requires newly purchased systems to 
be EAC certified 
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The Plan 

• Start with Premier Assure 1.2 counties 

- All Assure counties were mandated to upgrade 
to Assure 1.2 

- EAC certified system 

• Don't swallow the entire elephant 

- GEMS servers only 

• Work with the EAC and vendor to 
understand what certified configuration is 
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The Process_ 

• Parse the vendor provided verification tools 
(uneditable pdf) to a useable format (raw 
text) 

• Run SHA1 hash check on GEMS program 
directory using portable COTS software 

• Confirm hash values match EAC 
certification through the use of hash 
comparison software 

• Identify Windows 2003 Server security 
configuration (user accounts, rights, 
running services) 
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The Results 


• Hash checks of GEMS servers show no 
differences across counties 

• Physical checks of the systems show no 
differences across counties 

• The system setup and rights vary greatly 
from one county to the next 

- Possibly uncertified configuration 

- Possibly significantly less secure 
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Fairfield 


Crawford 


Darke 


Defiance 


Fulton 


Gallia 


Greene 


Guernsey 


Hancock 


Highland 


Adams 

Ashland 


Belmont 


Sutler 


Carrol 


Coshocton 


Harrison 


Hocking 


Holmes 


Huron 


Jackson 


Hardin 
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Board of Flections j Secretary of State 
Information Technology Security Review 
Directives 2008-56, 73 



Storage Requirements ot Election Equipment (2008-56) 

Climate controlled location 

1 


Security Requirements (2008-56) 

Access to secure rooms kept to minimal number of privileged! BOE personnel 




Minimum Access Control Requirements (2008-56) 

Entrv/Exit Jog 


Security Requirements Tabulation Server Room (2008-56) 

Access to secure rooms kept to minimal number of privileged BOE personnel 


Room secured by a double lock system 

■ 

Minimum Access Control Requirements (2008-56) 

Entry/Exit log 


Password Management on Tabulation Server 

BIOS Password In place, Split R/D 


Windows Account Password, Split R/D 



Password Complexity (2008-731 
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State Goals 


• Establish the baseline configuration for 
each voting system, regardless of vendor 

• Baseline includes tabulation software and 
system configuration 

• Confirm deployed systems match that 
configuration 

• Work with vendors and jurisdiction to bring 
systems into proper configuration 
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State Conclusions 


• Provided validation tools did not include 
mechanism for comparison, nor a simple 
way to compare only static files. 

• Produces additional overhead in 
confirmation process. 

• Hash codes must be manually transcribed 
for visual and/or text comparison 

• An automatic utility would be preferable: 
faster and more accurate 
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EAC Conclusions 



• The tools were not a form that could 
readily be used, (e.g., received in pdf file 
format) 

• The state would need to procure a COTS 
hashing tool to compare against the PDF. 

- No automatic comparison. A person would 
have verify each hash by sight or manually 
transcribe the values. 

• Poor quality hardware pictures requiring 
special tools and knowledge. 
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EAC Conclusions 


• The EAC's program did not require the 
tools to be checked for functionality or 
usability by any parties. 

• Vendors basically submitted whatever they 
wanted under the heading of “System ID 
Tools". 
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EA C & S ta te Nex t Ste p s_ 

• Validate the voting systems (not just 
servers) 

• EAC work with state and jurisdictions to 
understand their needs 

• Talk with other states to learn their process 

- are there other reasonable paths? 

• Work with vendor to understand differences 
and certified configuration 
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